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A TECHNIQUE FOR PROVIDING CONTINUOUS 
SPEECH RECOGNITION AS AN ALTERNATE INPUT DEVICE 
TO LIMITED PROCESSING POWER DEVICES 



PROVISIONAL APPLICATION 
5 This application claims the benefit of U.S. Provisional Application No. 60/202,101, 

entitled, "A TECHNIQUE FOR PROVIDING CONTINUOUS SPEECH RECOGNITION 
AS AN ALTERNATE INPUT DEVICE TO LIMITED PROCESSING POWER DEVICES 
SUCH AS PDAS," filed May 4, 2000, by James L. Keesey et al., attorney's reference number 
STL9-2000-0052US1, which is incorporated by reference herein. 

10 FIELD OF THE INVENTION 

This invention relates in general to a computer implemented system, and more 
particularly, to providing continuous speech recognition as an alternate input device to limited 
processing power devices such as personal digital assistants (PDAs). 

BACKGROUND OF THE INVENTION 

15 A personal digital assistant (PDA) is a handheld device that combines computing with 

other features, such as telephone and/or networking connections. Many PDAs are used as 
personal organizers and include calendars, e-mail systems, and word processors. Input is 
typically entered into a PDA via a stylus, rather than through a keyboard or mouse. A stylus is 
a "pen-like" object that is used to write data on a screen, such as a digital tablet. The stylus has 

20 an electronic head that is used to touch the digital tablet, which contains electronics that enable 
it to detect movement of the stylus and translate the movements into digital signals for the 
computer. 

Some PDAs incorporate handwriting recognition features that enable users to 
"handwrite" data onto the screen using the stylus. However, conventional handwriting 
25 recognition systems sometimes misinterpret written data, which requires users to carefully 
review and correct written data. 
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PDAs have become very popular and are increasingly being used by a wide spectrum 
of people. Unfortunately, these small devices have limited memory, a small display, and 
operate at slow speeds. Additionally the use of a stylus to enter data prevents some disabled 
persons from using PDAs. 
5 Thus, there is a need in the art for an improved technique of inputting data into a device 

with limited resources. 

SUMMARY OF THE INVENTION 
To overcome the limitations in the prior art described above, and to overcome other 
limitations that will become apparent upon reading and understanding the present 
10 specification, the present invention discloses a method, apparatus, and article of manufacture 
for a technique for providing continuous speech recognition as an alternate input device to 
limited processing power devices such as personal digital assistants (PDAs). 

According to an embodiment of the invention, a technique for data entry at a device is 
provided. Initially, voice data is received at the device. The voice data and a device identifier 
15 are transmitted to a computer. At the computer, the voice data is translated to text. Next, it is 
determined whether to filter the translated text. When it is determined that the translated text 
is to be filtered, a filter is applied to the translated text. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Referring now to the drawings in which like reference numbers represent 
20 corresponding parts throughout: 

FIG. 1 is a schematic that illustrates a hardware environment of an embodiment of the 
present invention. 

FIG. 2 is a schematic that illustrates a CSR System 212 and its environment in one 
embodiment of the invention. 
25 FIG. 3 is a flow diagram illustrating a process performed by the CSR System 212 in 

one embodiment of the invention. 
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DETAILED DESCRIPTION 
In the following description of embodiments of the invention, reference is made to the 
accompanying drawings which form a part hereof, and which is shown by way of illustration 
specific embodiments in which the invention may be practiced. It is to be understood that 
5 other embodiments may be utilized as structural changes may be made without departing from 
the scope of the present invention. 

Hardware Architecture 
FIG. 1 is a schematic illustrates a hardware environment of an embodiment of the 
present invention, and more particularly, illustrates a typical distributed computer system using 

10 a network 100 to connect voice data input devices 102 ("clients") to a server computer 104 
executing computer programs, and to connect the server system 104 to data sources 106. A 
data source 106 may store, for example, user profiles that include voice print records. A 
typical combination of resources may include voice data input devices 102 that are, for 
example, personal computers or workstations, telephones or cell phones, or personal digital 

15 assistants (PDAs). A server computer 104 may be, for example, a personal computer, 
workstation, minicomputer, or mainframe. These systems are coupled to one another by 
various networks, including LANs, WANs, SNA networks, and the Internet. Some voice data 
input devices 102 (e.g., a personal computer or a personal digital assistant) and the server 
computer 104 additionally comprise an operating and one or more computer programs. 

20 The server software includes a Continuous Speech Recognition (CSR) System 110, 

which comprises one or more computer programs for converting voice to text, filtering the 
text, and converting the text to an appropriate format. The server computer 104 also uses a 
data source interface and, possibly, other computer programs, for connecting to the data 
sources 106. The voice data input devices 102 are bi-directionally coupled with the server 

25 computer 104 over a line or via a wireless system. In turn, the server computer 104 is bi- 
directionally coupled with data sources 106. 

The operating system and computer programs are comprised of instructions which, 
when read and executed by the voice data input devices 102 and server computer 104, cause 
the devices and computer to perform the steps necessary to implement and/or use the present 
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invention. Generally, the operating system and computer programs are tangibly embodied in 
and/or readable from a device, carrier, or media, such as memory, other data storage devices, 
and/or data communications devices. Under control of the operating system, the computer 
programs may be loaded from memory, other data storage devices and/or data communications 
5 devices into the memory of the computer for use during actual operations. 

Thus, the present invention may be implemented as a method, apparatus, or article of 
manufacture using standard programming and/or engineering techniques to produce software, 
firmware, hardware, or any combination thereof. The term "article of manufacture" (or 
alternatively, "computer program product") as used herein is intended to encompass a 
10 computer program accessible from any computer-readable device, carrier, or media. Of 
course, those skilled in the art will recognize many modifications may be made to this 
configuration without departing from the scope of the present invention. 

Those skilled in the art will recognize that the exemplary environment illustrated in 
FIG. 1 is not intended to limit the present invention. Indeed, those skilled in the art will 
15 recognize that other alternative hardware environments may be used without departing from 
the scope of the present invention. 

Continuous Speech Recognition System 
In one embodiment, the present invention provides a Continuous Speech Recognition 
(CSR) System. The CSR System enables devices with limited processing power to provide 

20 continuous speech recognition. That is, most handheld devices (e.g., PDAs or cellular phones) 
do not have the processing power to perform continuous speech recognition. This, combined with 
their small size, forces users to use a stylus to peck at an input area, which makes these devices 
extremely difficult to use by disabled persons. It also prevents individuals from quickly taking 
notes, updating calendars, or sending e-mail. 

25 With the CSR System, inputting information into the device becomes as simple as 

speaking. The CSR System could conceivably remove the need for a tactile input device. The 
CSR System also allows for devices that are too small to have an input pad or screen, such as wrist 
worn devices, to be used as input devices. 
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FIG. 2 is a schematic that illustrates a CSR System 212 and its environment in one 
embodiment of the invention. The CSR System 212 is at a voice recognition server 210. The 
CSR System 212 establishes a synergistic relationship between one or more client devices (limited 
processing power devices) and one or more voice recognition servers. For ease of illustration, one 
5 client device 200 and one voice recognition server 2 1 0 are depicted. The client device 200 is able 
to record and/or relay speech. The CSR System 212 comprises voice to text software 214 and text 
filtering and transformation software 216. 

Generally, the client device 200 captures speech and sends it to the voice recognition 
server 210 for translation and transformation. The voice recognition server 210 sends the 
1 0 transformed information back to the client device 200, which then incorporates it into its target 
application (e.g., a calendar, e-mail, or notes application). 

Prior to using the CSR System 212, a user submits information to the voice recognition 
server 210. The information comprises a user profile 218 that is stored in a data store. The user 
profile includes a "voice print" associated with the way a user speaks, information about one or 
1 5 more target applications that are to receive data, one or more client device ("unit") identifiers ("ids 
") that identify a particular device used by the user, and contact information for the user, including 
an e-mail ("electronic mail") address. 

Initially, a user records speech that is stored as a voice print at the voice recognition server. 
For example, each user may be asked to speak particular text, such as a paragraph of a book. The 
20 spoken text is a voice print. Each user speaks a little differently, with slightly different pauses and 
intonations. Thus, a voice print may be used to identify a user. Additionally, the voice print is 
used by the CSR System 212 to better convert voice to text. 

Once the user profile 218 is stored at the voice recognition server 210, a user can input 
voice data into a client device 200 by speaking into a speech recorder/relayer at the client device 
25 200. The user speaks keywords and other speech. The keywords indicate to the CSR System 212 
that particular types of information follow. Sample keywords include, without limitation, the 
following: CALENDAR ENTRY, DATE, TIME, SEND NOTE, ADDRESS ENTRY, NOTEPAD 
ENTRY. To schedule a meeting in a calendar application, a user might speak the following into 
the client device 200: CALENDAR ENTRY DATE December 1, 2000 TIME 10:00 a.m. 
30 SUBJECT meeting on projectx. 
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The client device 200 uses this voice data to generate a speech packet that consists of the 
voice data (e.g., the phrase), data appropriate to the target application (e.g., the calendar 
application), and a unit id ( client device identifier). The client device 200 sends the speech packet 
to the voice recognition server 210 over any available communication system, such as cellular 
5 modem and/or an Internet connection. 

The voice recognition server 210 receives the speech packet, extracts the unit id, and uses 
it to retrieve the user's voice print from a data store. The voice to text software 214 uses the voice 
print to translate the voice data in the speech packet to text. This results in "translated text." 

Next, the text filtering and transformation software 216 attempts to extract one or more 

10 keywords from the translated text. In one embodiment, the one or more keywords are expected 
to be at the beginning of the translated text. If no keywords are found, the CSR System 212 
returns the translated text to the client device 200 by, for example, e-mail. On the other hand, if 
one or more keywords are extracted, the CSR System 212 identifies and retrieves a transformation 
filter ("filter") 220 to be used to format the translated text to a particular format (e.g., specific to 

15 a particular application and/or a specific device). For example, if the one or more keywords 
indicate that the voice data is associated with a calendar application and represents a CALENDAR 
ENTRY, the text filtering and transformation software 216 determines that a transformation filter 
is to be used and retrieves a calendar filter from the transformation filters 220 to format the data 
to be sent to a client device 200 as a calendar entry. The formatting will not only format the 

20 translated text for aparticular application (e.g., a calendar application), but the formatting will also 
format the translated text for a particular client device 200 (e.g., a particular brand of a PDA). 
Then, the CSR System 212 returns the filtered text to the client device 200 using an appropriate 
communication channel (e.g., via an e-mail over a cellular modem and/or the Internet). 
The client device 200 receives the translated and transformed speech packet and routes it 

25 to the targeted application (e.g., a calendar application) for processing. 

If the client device 200 is a cellular telephone, a user can input speech via the cellular 
telephone. The speech and unit id are sent to the voice recognition server 210. The CSR System 
212 at the voice recognition server 210 converts the voice data to translated text, applies a filter 
if that is appropriate to generated filtered text, and returns either translated text or filtered text via 

30 e-mail to the user's device, as specified in the user profile. 
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Thus, with the CSR System 212, to schedule a meeting in a calendar application, a user 
might speak the following into the client device 200: CALENDAR ENTRY DATE December 1, 
2000 TIME 10:00 a.m. SUBJECT meeting on projectx. Then, the CSR System 212 formats the 
voice data as a calendar entry, ready to be incorporated into a calendar. On the other hand, in a 
5 conventional system, a user would have to open the calendar application, locate the date and time, 
and type or write in the subject information. On a PDA, this typically requires use of a stylus, 
which is difficult to use for many people, especially those who are disabled. Additionally, it is 
not possible with conventional systems to generate a calendar entry with just a cellular phone. 
FIG. 3 is a flow diagram illustrating a process performed by the CSR System 212 in one 
1 0 embodiment of the invention. It is to be understood that, in one embodiment, the CSR System 212 
encompasses both the voice to text software 214 and the text filtering and transformation software 
216. 

In block 300, the CSR System 212 receives a user profile 218, including a voice print and 
a unit id, and stores the user profile 2 1 8 at the voice recognition server 2 1 0. In block 302, a client 

1 5 device 200 receives voice data and forwards the voice data and a unit id to the voice recognition 
server 210. In block 304, the CSR System 212 at the voice recognition server 210 retrieves a 
voice print for the user based on the unit id. In block 306, the CSR System 212 converts the voice 
data to text using the voice print, resulting in translated text. In block 308, the CSR System 212 
determines whether a filter is to be applied. If so, the CSR System 212 continues to block 312, 

20 otherwise, the CSR System 212 continues to block 310. In block 310, the CSR System 212 
returns translated text to the client device 200. In block 312, the CSR System 212 selects and 
retrieves a transformation filter 220. In block 314, the CSR System 212 applies the transformation 
filter to the translated text, resulting in filtered text. In block 316, the CSR System 212 returns 
filtered text to the client device 200. In one embodiment, the CSR System 212 returns the filtered 

25 text to an application at the client device 200. 

Conclusion 

This concludes the description of embodiments of the invention. The following describes 
some alternative embodiments for accomplishing the present invention. For example, any type 
30 of computer, such as a mainframe, minicomputer, or personal computer, or computer 
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configuration, such as a timesharing mainframe, local area network, or standalone personal 
computer, could be used with the present invention. 

The foregoing description of embodiments of the invention has been presented for the 
purposes of illustration and description. It is not intended to be exhaustive or to limit the 
invention to the precise forms disclosed. Many modifications and variations are possible in light 
of the above teaching. It is intended that the scope of the invention be limited not by this detailed 
description, but rather by the claims appended hereto. 
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WHAT IS CLAIMED IS: 



1 LA method of data entry at a device, comprising: 

2 receiving voice data at the device; 

3 transmitting the voice data and a device identifier to a computer; and 

4 at the computer, 

5 translating the voice data to text; 

6 determining whether to filter the translated text; and 

7 when it is determined that the translated text is to be filtered, applying a filter to 

8 the translated text. 

1 2. The method of claim 1 , further comprising storing a user profile in a data store 

2 connected to the computer. 

1 3 . The method of claim 2, wherein the user profile comprises a voice print. 

1 4. The method of claim 3, further comprising translating the voice data to text using 

2 a voice print. 

1 5 . The method of claim 1 , wherein determining comprises extracting one or more key 

2 words from the translated text. 

1 6. The method of claim 5, wherein a filter is selected based on one or more extracted 

2 key words. 

1 7. The method of claim 1, wherein applying the filter comprises formatting the 

2 translated text. 

1 8. The method of claim 7, wherein formatting comprises formatting the translated text 

2 for an application. 
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1 9. The method of claim 7, wherein formatting comprises formatting the translated text 

2 for the device. 

1 10. The method of claim 1 , further comprising returning translated text to the device. 

1 11. The method of claim 1 , further comprising returning filtered text to the device. 

1 12. The method of claim 11, further comprising retarning the filtered text via an 

2 electronic mail message. 

1 13. The method of claim 1, further comprising returning data to a device other than the 

2 device at which voice data was received. 

1 14. An apparatus, comprising: 

2 a device for receiving and transmitting data; 

3 a computer having a data store coupled thereto, wherein the data store stores data, 

4 connected to the device; and 

5 one or more computer programs, performed by the computer for: 

6 receiving voice data and a device identifier from the device; 

7 translating the voice data to text; 

8 determining whether to filter the translated text; and 

9 when it is determined that the translated text is to be filtered, applying a filter to 
1 0 the translated text. 

1 15. The apparatus of claim 14, further comprising storing a user profile in a data store 

2 connected to the computer. 

1 16. The apparatus of claim 1 5, wherein the user profile comprises a voice print. 
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1 17. The apparatus of claim 16, further comprising translating the voice data to text 

2 using a voice print. 

1 18. The apparatus of claim 14, wherein determining comprises extracting one or more 

2 key words from the translated text. 

1 19. The apparatus of claim 18, wherein a filter is selected based on one or more 

2 extracted key words. 

1 20. The apparatus of claim 14, wherein applying the filter comprises formatting the 

2 translated text. 

1 21. The apparatus of claim 20, wherein formatting comprises formatting the translated 

2 text for an application. 

1 22. The apparatus of claim 20, wherein formatting comprises formatting the translated 

2 text for the device. 

1 23. The apparatus of claim 14, further comprising returning translated text to the 

2 device. 

1 24. The apparatus of claim 1 4, further comprising returning filtered text to the device. 

1 25. The apparatus of claim 24, further comprising returning the filtered text via an 

2 electronic mail message. 

1 26. The apparatus of claim 14, further comprising returning data to a device other than 

2 the device at which voice data was received. 
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1 27. An article of manufacture comprising a program storage medium readable by a 

2 computer and embodying one or more instructions executable by the computer to process data 

3 entered at a device, comprising: 

4 receiving voice data at the device; 

5 transmitting the voice data and a device identifier to a computer; 

6 at the computer, 

7 translating the voice data to text; 

8 determining whether to filter the translated text; and 

9 when it is determined that the translated text is to be filtered, applying a filter to 
1 0 the translated text. 

1 28. The article of manufacture of claim 27, further comprising storing a user profile 

2 in a data store connected to the computer. 

1 29. The article of manufacture of claim 28, wherein the user profile comprises a voice 

2 print. 

1 30. The article of manufacture of claim 29, further comprising translating the voice 

2 data to text using a voice print. 

1 3 L The article of manufacture of claim 27, wherein determining comprises extracting 

2 one or more key words from the translated text. 

1 32. The article of manufacture of claim 3 1 , wherein a filter is selected based on one or 

2 more extracted key words. 

1 33. The article of manufacture of claim 27, wherein applying the filter comprises 

2 formatting the translated text. 
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1 34. The article of manufacture of claim 33, wherein formatting comprises formatting 

2 the translated text for an application. 

1 35. The article of manufacture of claim 33, wherein formatting comprises formatting 

2 the translated text for the device. 

1 36. The article of manufacture of claim 27, further comprising returning translated text 

2 to the device. 

1 37. The article of manufacture of claim 27, further comprising returning filtered text 

2 to the device. 

1 38 . The article of manufacture of claim 37, further comprising returning the filtered 

2 text via an electronic mail message. 

1 39. The article of manufacture of claim 27, further comprising returning data to a 

2 device other than the device at which voice data was received. 
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A TECHNIQUE FOR PROVIDING CONTINUOUS 
SPEECH RECOGNITION AS AN ALTERNATE INPUT DEVICE 
TO LIMITED PROCESSING POWER DEVICES 



ABSTRACT 

A technique for data entry at a device is provided. Initially, voice data is received at the 
device. The voice data and a device identifier are transmitted to a computer. At the computer, 
the voice data is translated to text. Next, it is determined whether to filter the translated text. 
When it is determined that the translated text is to be filtered, a filter is applied to the translated 
text. 



STL9-2000-0052US1 



-14- 



IBMST 045161 

::ODMA\PCDOCS\DOCS\27230\2 



CO 

o 



CD 
O 




O 
O 

q: 
uj 

i 

UJ 
CO 





Illltlllttllllll 








\ 



o 



u. 



\ 



CO 

< *~ 
Q (O 

O 3 
> CL 




LU 
h- 
CO 
>- 
CO 

q: 

CO 

o 




Q § 

c « 

o « 

« < 
© 



RECEIVE USER PROFILE, 300 
INCLUDING VOICE PRINT / 
AND UNIT ID, AND * 
STORE AT SERVER 



RECEIVE VOICE DATA AT 
CLIENT AND FORWARD 
VOICE DATA AND 
UNIT ID TO SERVER 



302 



RETRIEVE VOICE PRINT 
FOR USER 



304 



CONVERT VOICE DATA TO 
TEXT USING VOICE PRINT, 
RESULTING IN 
TRANSLATED TEXT 



306 




NO 


310 







RETURN TRANSLATED 
TEXT TO CLIENT 



SELECT AND RETRIEVE 
TRANSFORMATION FILTER 







312 



APPLY TRANSFORMATION 
FILTER TO TRANSLATED 
TEXT, RESULTING IN 
FILTERED TEXT 



314 



RETURN FILTERED 
TEXT TO CLIENT 



316 



FIG. 3 



DECLARATION AND POWER OF ATTORNEY FOR PATENT APPLICATION 



Page 1 of 2 
DOCKET : STL 9 -2000-0052US1 



As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name; 

I believe I am the original, first and sole inventor (if only one name is listed below) or an 
original, first and joint inventor (if plural names are listed below) of the subject matter 
which is claimed and for which a patent is sought on the invention entitled 

A TECHNIQUE FOR PROVIDING CONTINUOUS SPEECH RECOGNITION AS AN ALTERNATE INPUT DEVICE TO 
LIMITED PROCESSING POWER DEVICES 

the specification of which (check one) 

X is attached hereto. 

was filed on 

as Application Serial No. 

and was amended on {if applicable) . 

x ,^ ereb y state that I have reviewed and understand the contents of the above identified 
^ff cif ication ' including the claims, as amended by any amendment referred to above. 

^Acknowledge the duty to disclose information which is material to patentability as defined 
Ml Title 37, Code of Federal Regulations, Section 1.56. 

Ijjiereby claim foreign priority benefits under Title 35, United States Code, Section 119 of 
any foreign application (s) for patent or inventor's certificate listed below and have also 
%f ntified below an Y foreign application for patent or inventor's certificate having a filing 
date before that of the application on which priority is claimed: 

PtlLor Foreign Application (s) Priority Claimed 

£*2ne Yes No 

•=4 (Number) (Country) (Day/Month/ Year Filed) 

I;;%ereby claim the benefit under Title 35, United States Code, Section 12 0 of any United 
States application (s) listed below and, insofar as the subject matter of each of the claims 
of this application is not disclosed in the prior United States application in the manner 
provided by the first paragraph of Title 35, United States Code, Section 112, I acknowledge 
the duty to disclose information which is material to patentability as defined in Title 37, 
Code of Federal Regulations, Section 1.56, which occurred between the filing date of the 
prior application and the national or PCT international filing date of this application: 



(Application Serial No.) (Filing Date) (Status) (patented, pending, abandoned) 

I hereby claim the benefit under Title 35, United States Code § 119(e) of the United States 
provisional application listed below: 

U.S. Provisional Application No. 60/202,101, filed May 4, 2000. 

I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 
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POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attorney (s) and/or 
agent (s) to prosecute this application and transact all business in the Patent and Trademark 
Office connected therewith. (list name and registration number) 

Romualdas Strimaitis , Registration No. 35, 697 , Prentiss W. Johnson , Registration No. 33,123 , 
Ingrid M. Foerster , Registration No. 36,511 , Timothy M. Farrell , Registration No. 37,321 , 
Christopher A. Hughes , Registration No. 26,914 , John E. Hoel , Registration No. 26,279 , Edward 
A. Pennington , Registration No. 32,588 , and Joseph C. Redmond, Jr. , Registration No. 18,753 , 
all of whom are attorneys with IBM Corporation; and 

Laurence H. Pretty , Registration No. 25, 312 , Robert A. Schroeder , Registration No. 25,373 , 
and Janaki Komanduri , Registration No. 40,684 , all of whom are attorneys with Pretty, 
Schroeder & Poplawski . 



Send correspondence to: 



Janaki Komanduri, Esq. 

Pretty & Schroeder, P.C. 

444 S. Flower Street, 19th Floor 

Los Angeles, California 90071 



D|rect Telephone Calls to: (name and telephone number) Janaki Komanduri, (213) 622-7700 



F||ll name of sole or first joint -inventor: James L. Keesey 




Date 



Residence: 867 S. Knickerbocker Drive, Sunnyvale, California 94087 



Citizenship: United States 



Post Office Address: Same 



Full name of second joint-invjen^dr : 



J. Wilmot 



Inventor ■ s signature : 




Date 



Residence: 3197 Vista Del Camino, Marina, Califprnia 93933 



Citizenship: United States 



Post Office Address: Same 
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